14 research outputs found
Efficient Estimation and Control of Unknown Stochastic Differential Equations
Ito stochastic differential equations are ubiquitous models for dynamic
environments. A canonical problem in this setting is that of decision-making
policies for systems that evolve according to unknown diffusion processes. The
goals consist of design and analysis of efficient policies for both minimizing
quadratic cost functions of states and actions, as well as accurate estimation
of underlying linear dynamics. Despite recent advances in statistical decision
theory, little is known about estimation and control of diffusion processes,
which is the subject of this work. A fundamental challenge is that the policy
needs to continuously address the exploration-exploitation dilemma; estimation
accuracy is necessary for optimal decision-making, while sub-optimal actions
are required for obtaining accurate estimates.
We present an easy-to-implement reinforcement learning algorithm and
establish theoretical performance guarantees showing that it efficiently
addresses the above dilemma. In fact, the proposed algorithm learns the true
diffusion process and optimal actions fast, such that the per-unit-time
increase in cost decays with the square-root rate as time grows. Further, we
present tight results for assuring system stability and for specifying
fundamental limits of sub-optimalities caused by uncertainties. To obtain the
results, multiple novel methods are developed for analysis of matrix
perturbations, for studying comparative ratios of stochastic integrals and
spectral properties of random matrices, and the new framework of policy
differentiation is proposed